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apStRACT : 

The increased attention +o measuring career skills 
has resulted-“in nore instrument: development, more testing of 
students, ‘and more.test administrators. There are three key areas of 
concern. The’ first area is that of identifying purpcses to be served 
by assessing career skills, Purposes include permitting descriptions 
of the current status of individuals or sroupa, /and evaluation 
(effectiveness, utilization in preqram improv nt, allocation ‘of 
resources, and -allocation of money). The second area is that of. 
becoming an informed user of career skills tests--Is the instrument. 
designddgfor the test audience?: Is the reading level appropriate?; 
Is the test reliable?:*Is Tt valid?; and Is it free of Segmal and 
cultural’ stereotyping? The third area concerns limitatiq ig test | 
design and use. There are two principal concerns about t design of 
‘‘paper and pencil: career skills asgessmef*+ materials. They are (1) © 
indirect measurement and the froblem of sampling in assessing career. 
skills, and,(2) the effects of verbal ability on student perf orma pce 
cn measures of career skills. Approprfate cameron should Be obser Ved - 
in nib canaan f oareer ski lis. oy 
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The Educational Resources information Center. on Adult, 
Career, and Vocational be career (ERIC/CE) is one of . 
sixteen clearinghouses tf nationwide information * 
system that is funded by ee National Institute of 
Education.*® One of the functions of the Clearinghouse 
is to ,interprét the literature”: t is entered_in the 
ERIC data base. This paper should be of particulaz-——— -— cae SEES 
interest to practitioners of career education, ee 
at the middle school and secondary school levels. 


The profession is indebted to “Arthur M. Kroll and Lind@& 
‘A. Pfister for their gcholarship in the preparation of 
wthis paper. Recognition also is due; Roger Aubrey, 


Joely Cowan, American Training and Research Associates; 
and Donald Fischer, the National Center for Research in | 
vocational Education, for their critical review of the 
manuscript-prior to its:final revision and publicatjon. 
Robert D. Bhaerman, Assistant Director for Career Edu- 
cation at the-RRIC Clearinghouse on Adult, Career, and 
Vocational Education,.coordinated the publication's 
development. Cathy Thompson assisted in the editing of 
the manuscript and Bonna Somerlott typed the final. draft. ’ 
a 
Robert E. Taylor / °. ff 
Executive Director 
’ The Nationa?P'Center for Research 
in Vocational Education. 
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The increased atténtion to measuring career skills has 
resulted in more instrument development, more testing ‘of 


key areas of concern., The first area is that of identifying 
purposes to’ be. served by assessing career skills. Purposes. 
include permitting descriptions of the current status of... 
individuals: or groups, and evaluation -(effectiveness, | 
utilization in program improvement, allocation’ of resources, 
and allocation of money) . The second area is that of’ 
becoming an informed user of career skills tests -- Is the 


-- etudents, pe more: test administrators. There are three: 


“instrument designed for the,test audience?;. Is the reading 


‘opriate?; Is thé’ test reliable?; . Is it valid?; 
ree of sexual and cultural stereotyping? The 

me ea concetns limitations in test design and use. 
There are ,two principal pens about the design of paper 
and pencil careeg skills assessment materials. They are 

(1) indirect measurement and the problem of sampling in 
assessing career skills, and (2) the effects of verbal 
ability on stYdent performance on meagures of career skills. 
Appropriate caution should be. observed in using tests of 
career skills. ° (CT) ° ; 
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a «The process of assessing career skills has received particular 
attention in the' 1970s with in@fteased\interest in tareer education 
and concerhs about ‘accountability. With the expansion of the e 
importance of career education -- as well as federal and state © 
‘funds for, financial support -- came the requirement for specific 
ways to eWaluate the impact of career education on students. 
Pressure began to mount for a clearer definition of*career educa- 
tions As Director of USOE's Office of Career Education, 
Kenneth B. Hoyt provided a ,definition by advocating a results- 
oriented approach-outlining specific learner outcomes. (He 
states: ’ 


Career education #eeks tq produce individuals 
whe, when they leave school (at any age or at any 
level) are : 
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Ne 


. - -Gompetent in the basic academic skills 
7 ; required for adaptability in ovr rapidly 
changing society; 


* equipped with good work habits;. 


’» equipped with a personally meaningful = ae , 
* set of work Walues that’ foster in them pt << , 
a desire to work; io - 
- ; ~e) . f ‘. . é 
© egt'ippea with career decision-making 4“ a { 


“skills, and job-getting skills; 
F \ » Vs : 


e . and understanding of educ tional-vocational 
| - Opportunities sufficient /for making sound 


- equipped with a q of, educhtsonalvocational. 
oe . ‘career decisivwns;. . x 
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ww 


* aware of mearis available to: them for ae 
7 * continuing cs a acta eaneaulony 


s 


- either ‘placed or actively ere plicanent 
« in a paid-occupation, in further education, © 
“* or in a vOcation consistent with their SC 
a Career decisions; se 7H 4 
-* actively Week nd to find meaning and 
meaning fulness through work in: PEOENG ES YS 
r use of leisure ‘time; 
*-aware of means. available to ‘themselves 
’ for changing. career options -- of societal 
and personal constraints impinging’on ? 
\ career alternatives’ ; 
(Hoyt, 1977, p. 35)- = 3 : 


: - , 8 
In essence, the career education movement has progressed to the 
‘point that a set of specific skills has been established -- skills 
that can be taught, learned, and Sener eee We refer to these 
skills as career skills. | ° 
The increased attention to measuring career skills has resulted 
in more instryment development; more testing of students, and,, 
of.course, more test administrators. The \purpose of this paper’ 
is to assist Peactitioners who select and use career skill, 
tests. “Three major topics are covéred: ‘identifying urposes to 
be pe ae by assessing career skills, points to keep in mind if 


you aré, to. become an informed user of tests of career skills, 
and limitations. in test design. with cautions against possible 
misus’. ‘ : oD $ 
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PURPOSES SERVED BY ASSESSING CAREER SKILLS ou = 


te is good practice to have your purposes clearly in mind when 
you administer any test of career skilids. Such purposes may 
be as ‘Varia as facilitating student development, valuating’ 
program effectiveness, or obtaining hase-line. data] for an 
experimental research project. If you haven't identified your 


| , : 


anticipated uses of 'the test results, you should delay administration 
of the materials. until you have planned more -carefully. Knowing why f 
. you are administering the tests, what ydu Hope to. achieve in terms. — 
of student development ‘or new sleeve ee how you will follow 
through in use of the results are important elements in planning. 


Why try to assess career skills? What possible purposes can be 
_ served, by such testing? Four such purposes are among the broad 
a goals of educational assessment in general: first, to find out 
A ; where individuals stand in their development of career, skills and 
knowledge; second, to identify areas where instruction or guidance - 
may be needed; third, to find out whether such .intervention is ° 
beneficial; and, fourth, ,to compare the-effectiveness of different ° 
forms of “instruction. ee if 
The first purpose is primarily descriptive. The creation of a 
yardstick for assessing career skills permits description of the 
current status of individuals or groups. From-such descriptions, 
the user of the yardstick can evaluate subjectively the quality 
of different levels of performance. / ; 
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ge "Once a’ measure is used: and an evaluation is’ made, a decision can 
¥ _ be reached Ags to whether or not intervention is warranted in the 
é skill development process. This decision, too, is subjective. 
Although the measure yields data, decisions.that involve interpre~ ‘ 
, tation’ of: the data rest~with the user. ~ - : Maly 
. . Oe is, . * "9 ‘ . 
. The decision to intervene|raises diagnostic questiohs. What _" 
- : areas:'of relative strergth and weakness have been identified? 
- Which individuals need hejp and what kind of help? ; How, do the 
\ needs .relate, to possible teaching methods? Can learning exper- 
iencesbe designed. to address individual student needs? 
The other-purpose involves evaluation. . How effective are part- 
ictlar instructional, strategies? Can assessment of them improve 
a career education program? 'Can direction be changed midway ‘ 
‘to head toward a more promising path Can the relative effective-- 
ness' of different educational programs be determined? Are students 
; -in the program’ developing different or better skills than students 
who are not? Is th ‘instructional program worth its costs in time 
-  \and other resources? Should available resources pa\ al located 
jaifferentiy? _ I» 
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Throughout their school careets, ¢tudents make many decisions, |! 


Vio 


and still other decisions are made by school staff about them. 
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_- Tt often-is said that™the better the Anformation. on which a decision 
a ° 


is. based, the’ more satisfactory the cision is likely:to be. 

Tests of career skills canndt make any of these decisions; they 
can contribute,. however, ta the -information base = ‘on which the ~- 
decisions are-made. “GeO a 
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BECOMING AN INFORMED CONSUMER. OF CAREER SKILLS TESTS 


> 
Once you have determined the purpoges for which you ree to use 
career skills tests, you can take a critical look at the 
assessment tobls that are available Sample sets of instruments 
are, almost always available from commercial publishers. In 
addition, two publications currently in press which wyll provide 


neoamatton specifically on cdreer skills ‘tests are: “Career 


Hduecatton Measures: (A Compéndtum of Evaluation Instruments 
(McCaslin et al.,in press) and A Sourcebook on Evaluating . Career 
‘dueatton (College Entrance Examination Board, in press). 


Get well acquainted aie the tests you are considering so that 

you will understand ‘the purposes for which they were*designed. 

Be sure that. the two sets of purposes do not conflict. Whether or 
not your expertise is in the area of meastrement, you’ can become 


‘ ay informed’ consumer of: career skills testing materials by keeping, 


the paarenesee questions in mind as you evaluate each test: ‘ 


“dy. A&B the i sininent destgned forethe audtence, you plan 
test, Different measures are designed for different popula- 
tions. Check to see-what audience(s) the test has been adminis-. 
ted to. How does it compare to your own test population. Hoty 

aia the field trial.population perform on the test? 


, You may’ decide to usé a particular test with a ‘population 


different from that, of the test developer. If you are using the 
test as an instructional or guidance tool, , the age or grade devel 
of the measure's focug may not limit your decision. Howevér, if 
you plan to use the test as an evaluation tool and the test norms. 
Or other technical data as benchmarks to compare student perfor- © 
mance, then your audience should, aes match * that used by the 
test developer: - 


a 


‘2. 


Does the test bare wate the ponbent 
proposed test. takers|? 


As career’ skil 


ls ted 


belting taught to the’: 
;are. often used to 


valuate the effecti eness Of career, education’ programs., it is 


o 


‘Pessential- that you 


~goincide with the goals of your program. - 
‘ -content students are beirig taught? . 


all the questions i 
student to do. 


it? 


n 


8 
» °' tested? Readability 
ie selecting career Sk 


ake sure-that the 


I 
a career = skills 


Con ider each Blade carefull 


a 


Is’ Phe reading level appropriate -for the 


is always a fact 
lls tests. After 


Objectives of the measure 
Does {it measure the 


f possible| work through 
test as yd ee a 
each. . 


Can 


Is the item med suring something that is. cove éd in your 
‘4 school’ Ss ——— edugatibdn. or career guidance PFOS am?’ 


~ 


udienee :to be 
or td be considered, when 
all, you want\ to agsess 


growth in some area/of career development? not ,find\out how well, 


someone reads. Some 


analyses of*their mé terial-tn their technical manuals. 
that do not, you may wish to have the materials examined by a” 
reading specialist - in your. agency. 
of career development --.e.g.,.apprenticesHip, personhel, 
Dictionary of Occupational, TLGles 5: 
is replete v with some what difficult words. 


competent in career 
have am understandi 
effectiveness of an 
the’ students you ple 
_ their feedback. 
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- 4. What ts 
redial lity. should 
by ithe .test’ develop 

. Gefined as thé exte 
results. Reliabili 

. which indicates the 
“game score From one 
another. | 


0 and l, 


the 


commercial tést 


Oe 


g.of these terms. 
instrument is to 
n to test.* Seelh 


e provided in’ the 
soa The reliabili 


y may be express 
lextent to which — 
administration of 


If a measure was administer 
the same measure given ‘to the sane gx 
the two sets: of scores correlated 

goefficient would be an index of: the measure's-reliabilit 
reliability coefficient “is expressed jas a mumerical value 
although in practice it is u 


for measures of individual, per formanc 


coefficient is to l, 


the more reliabl 


rc 
| oe 
ed A 
J fo 


the’ resulting correlat 


developers report readability + 
For those 


t 


Remember that th . vocabulary 


hee lanka k Handbook bata 
Yet, for s udents to be 


* The best test of the’ 
bw" they oe it to 4 couple: of” 
w ehey, perform and ask hice 


pa 


technical manual 
tyof.a measure may 
as a correlation: oef ficient, 
n individual wquld 


ed to a group of ind 
oup several days lat 


idageere between .65 and 
The: closer the re iability 
@® the measure. 


o 


_interest. 


%, "4 3 
oe ye a 
oy . . . + ‘ 
. Often, «4h ‘praca reliability, ccefficfents’ hae hot derived by** | Oe 


ig tést-retest “technique because it. is nOt possible’ to, overcome 


the, effects. of: practige on “the; ‘second administration, ang -for to : 
other practical reasons. nstead;, the reliability coefficient | is” “- 
“usally ‘edmputed by analyzing statistics for indiyidual, questions 7 
from ‘Single. administration of, the instzument; this method , 
yields" ‘an index o£ internal consistency. Such indexes «answer the 
‘questions. To fihat @e@gree are the Questions on the measure inter- 
relat A high rplationship suggests that most of 'the: questiony 


— mlasuring Ene o trait or the: some Come rnaer On of t¥aits. 
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5. “What te the test's validity? *Validaty considera siogs. a aes 

too, are important “iin making.“decisigns about whether. or. no Oo. ioe 
use a career skills test. A measure may be.used in thany diff. 
erent. ways, apd validity.as a concept me enacts fay of a: measure | 
rather than to the measure itself. . Some measur@s‘thay be used 

for describing how much a student has ‘learned identifying: ee 
students for future participation ina particular skill-building 
,experience, or for deciding which: of several instructiondl approaches, 
designed to develop career skills is most effective. Because each of 
these applications is based on a different interpretation, the 
evidence ‘that justifies one application may have dittle relation 
to*another. Validation thus ‘calls for an integra ion of many 

types of eu idenceé “ae 

Three oe of validity may be distinguished: content, construct, 
and criterion-related. Content validity is evaluated by showing 
‘how well the content of the. measure samplés .the subject matter 
or situations about which conclusions are to be drawn. Construct * 
validity is evaluated by determining the degree to which certain 
explanatory concepts account for performance on &he measure. 
Criterion-related validity is evaluated by comparing scores, or 
predictions made from them, with another variable considéred to 
provide a direct eiale the. characteristia or behavior of 


a 


Find out what eypes of validity studies have been senauaesd using 
the test. You may decide to conduct your own content validation 
to decide the degree to which the test is: appropriate fore the 


“your particular ae re 
6. Ig the test free of sex and cultural stereotyping? < 
Goals in career education have stressed the inelusion of all e 


individuals in program efforts. You should examine the: 
materials carefully to determine bath obvious gnd_ subtle dis¢riminatory 
statements. Do the items steréotyp either sex into any particular 


roles? Do ‘they represent a multiplicity of cultural backgrounds? 


i 


‘ a % : ¢ 
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"In interpreting gest resulta, be sensitive to the fact that-the . “ 


. . ‘ty “performance on English-language measiires bf students whos¢ native, 
» 4 Janguagé is other than English may “be, négatively influenced, i? 
‘especially by low readirfg ability, or by Mack: of familiarity with /™» 
certain career-related: ‘terminolagy.- Similarly, test, results . 

‘ arg often reported by sex. In general it wakes little -sense to 
“.compare female and male group performange unless the results are 

to be uséd*in planging’ specific skill~b ilding experiences or ‘in 
condficting f 3 research study. Pee ge pS es ; 28 

° N : gt : aid ee ms y 4 
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LIMITATIONS IN DESIGN AND USE OF TESTS OF CAREER SKILLS 7 


: 4 Zz * : : i 
aes - “In the history of educational noasukdibent: tests of career skills 
are relative newcomers. The various commercially.available tes#s 
‘vary considerably in’theit) theoretical: and conceptual foundations 
and remain imperfect -instraments that must be: used with. full 
recognition of their’ Yimitations in design and intended use... 
Two principal concerns about the design of paper-and-pen¢}1 .career © 
skills assessment materials are: (1) indirect measurement and , , 
the problem of sampling. in ‘assessing career skills; and (2) the . ° 
effects’ of verbal ability on’ student performarice on measures 
Pe (- iof career skillf. a @ 2 “a . ao, 
“4 INDIRECT MEASUREMENT AND THE,- - 
PROBLEM OF SAMPLING ; 
" IN ASSESSING. CAREER SKILLS a - | 
: . ‘ : : . & , 
When standardized tests are wea in.an attempt to assess absttact 
characteristics like career skills, it’is important to have a 4 
ear appreciation of precisely what is being attempted. Such 
intangible elements of cognitive, functioning qannot be measured 
directly. by "taking a reading" on some standard measurjng device 
and recordihg the observed measuf#ement. .Each test designed to 
assesg: career skilts is an indirect measure that involves a 
series’ of readings (one for -every item in the te&t) of various 
_° indicators of the, skill or knowledge to be measured. The 
measure ,is "indirect! in the sense that it assesses 4. person's 
responses to a set.of structured bxercises rather than directly 
. assessing the person's behavior in' real-life situations. Something 
about the skid} iteeélf must then be inferted from the information 
thatehas been collected from the indicators of that skill. — 


ay 


, 


so 


Many things *cannat’ be measured directly. When’ we’ want to know, 
how cold itis as ors, we do not ,megsuxe the cold ‘(or nat | 
a oc 


itself. We know’ that’ hegt and cold make ‘things: expand and 
“contract. S@ we-use-a device like a colutn of mercury in =) 
‘ thermometer and ‘observe | and fluctuation to finfer how 
cold it is.- Such things as at, light,, electricity, attitudes ,i 


-errotionsy or abilities cannot be measured directly. The presence 


or absence of .such things, or.their amount or quality, is deter- 


_) ae by HHECE RCE’ . ‘7 : 


‘gomething else. A person's honesty,’ or .aggressiveness, * 
Or.loneliness could not be measured, but we can inf 
their presence, ‘or: quantity, by observing specific 
reactions and behaviors which we know are related. to- 
honesty, or aggressiveness’, or loneliness. ¢ ‘ 
Indirect measurement is a perfectly aaa ; 
. procedure and may. bé seen in fields other than 
-education.: For example,-.the amount of acid insa. es 
substance may’ be measured by the amount of base : — 4 
, required to neutralize the acid. Using a similar” 7 
: indirect .process), we can.measure a pwpil's 
ay peel gence by observing ‘certain kinds of behavior 
‘which. we assume are the “ror Ge the development 


‘We measure the effect that these iene, ‘have va) eee 


of his mental abilities. For (example, we might 
oe ere the.development of a p&pil's memory by 
serving his ability to memorize. We might 
estimate a pupil's reasoning ability by observing 
his ‘performance if solving’ problems,’ making 
_ generalizations, and discriminations. 
., (Using Teste in the Schoots,: 1962, ’p.5) 
a ~ . 


* 


In a way, a career skills measure is a ‘bi like radar, through 
which observations of a Series of "blips" ‘on a screen\are used ? 
to infer the characteristics of some unseen object.: Both: are, 
examples of indirect measurement which involves the interpretation 
of observations. However, the difference between radar and career 
skills measurement is very great. Radar is applied to physical 
objects or phenomena where clearly established Beene a of 
physical science permit very accurate interpretation on the 
observations, whereas care&r- s ills measurement is applied to 
cognitive processes’ where the principles are not yet clearly 
established,and interpretations are still probabilistic. ' 


Such Yindirect measurement “plices. ‘a considerable burdetii dn, !the 
‘+! set Of paper-and~pencil test questions from which an dndividual's 
-career skills and knowledge ‘must be inferred. If thé questions 
Or excercises are ‘representative of the univer rpe '@f Greer ‘skills 
and knowledge, then performance on them:will yield accurate and 
useful information. .However, to the extent that the exercises*’ — ° 
are not representative, any inference drawn from the’ performance 
data about: an individual's career skills may" "be subject to error... mo, 
How representative of a universe of careag behavior are the items ‘ 
‘that comprise ‘a given test?’ Is individual performance on a given’ 
sample from the universe_@f possible items indicative‘ of’ that y, 
individual's potential performance on other item samples drawn a oa 
- from the ‘hypothetical pool of itemg that comprise the content ie 
domain? Can one generalize about individual's competence on 
, _ +the basis of -a given pone e. © ./e 


« 


Each set of ‘questions “that comprises one of the commercially~ ’ 
available measures of career skills assess. only a sample of, the 
intricate complex of abilities and functions which dre. commonly, 
included under the term "career. development." In many situations 
it is both difficult and impractical to assess or evaluate ‘things 
in their entirety. .As.a conse uence, most evaluations are ‘based 
on samples. It yould be imposs ible, for example, to test’ for 


| pollution all of the water in a)\city' s water supply. ‘ A few ‘small y : 
samples are taken and tested, a the’ result is used to estimate - 
the quality of all the water. . a: ‘ 


-are conducted uging samples of only a few drops. jA winemaster 
judges the quality of thousands of Jiters of wine by periodic . 
tasting. -A manufacturer of prescription drugs evaluates thé ; 
content and purity of a product by withdrawing and testing the 
chemical composition of only a small sample of the continuing 
output. The butterfat content of a dairy farmer's milk’igs 
. determined by taking a test tube sasample of milk withdrawn from 
the farmer's refrigerated storage tanks. In each case, the 
quality of the total amount ‘is. estimated on the ‘basis of the _ 
sample. 


Many "tests" “aging. kind have become commonplace. Blood wecte. 


. 


hoo- The same process can ba d to assess career ygekills. From the 
entire complex of skills da knowledge, we evaluate somé tiny 
samples, and from the results of such measurements we attempt, to 
. estimate an individual's functional competency in career~ related 
y endeavors. The following figure illustrates the process of 
Samp hing: from the total universe of career skills. 


, 
at, . : 
ca “a 


The adaptation of Lewin's (1936) topological representations of 
_psychdlogical structures presented in, the illustration is useful 
in. understanding »the nature of career na pemigs The major sub- — 
divisions of the: career skills universe’ may be: indicated by the’ ; 
thick lines. . Some areas are less clearly differentiated, since - | ? 
the skills involved may be shared. , Samples may be gaa from a 


a 


skills subdivision, ewe though each of t ese selected ills : 7 yo 
“gis so comprehensive that we cannot hope to. assess all of it. So, / 
* - in turn, each subdivision is divided into smaller skill areas, 


and an attempt is made to sample from each. As a consequence, 

any career skills test can actually assess only a few samples ~ _ 

from the various sectors of a person"s total universe Of career- _ 
. related skills. bi * 


. 


Sampling problems arise not only from the imperfectly defined and ; 


; described universe from which they are drawn, sbut also from 
7 inability within a relatively short®fleasure to sample both ‘ 7 P 
widely across skill areas and in depth within a skill area. vs. 
Grpbach and Glaser (1957) discussed this sampling dilemma in the 
ve S° Cal unication engineering terms’ of bandwidth and fidelity. 


“ge. test degisnee and the user oft AES oa me 4 
. pequentlywhave to choose between’ Gax jeful estima 
tion of a single variable and more cursory okplor-"." 


Mos ae ation of many separate variables. ,T¢sts may 
ny on a... to yield separate gcores on a number 
; of diverse, internally homogeneous scales, or to 
provide a single measure loaded with the general 
‘4 _ factor underlying items... The dilemma may be —~ 
. - “described in the language of the communications © 
a“ engineer as a ‘choice between "wideband" and ; 
_ "narrowband" tests. In using a particular channel, 
igh such:-as a tele — wire, ona may either crowd many 
2 messages into period of time, or give a single 
7 message slowly and repetitively... The former, more 
yi ‘ varied message has greater “bandwidth. * "The wide- 
: band signal transmits axe informat}on, but, the — 
clarity or dependability 
is less than for the narrowband signal except under 
ideal communication conditions. Ratton” Fors ; ‘can 
seriously confusé the. wideband signal; this is 
v. spoken of as a’ lack of’ fidelity. - The tester Ss 
situation is analogous. If he con entrated on 
facts-relevant to a single decisiony “he gets a 
~ ?much more dependable answerf##han if he spreads 
‘bis effort. But by-concerntrdting, he leaves all 
“his other questions to be answered on the bases of 
/ chance alone. - : ; a 
_ This suqgests that in any decision situatfon 
e thére is some ideal compromise between variety of 
'  jnformation (bandwidth) and thoroughnéss of testing 
to obtain more certain information (fidelity)... 
(p. 90) : : 


! 


<*> 


/ 


The sam ling’ probigh & in assessing career skills, then, is ane 
representative tasks that will require dis 

use of a reasonable variety of the kn wledge and skills in 

gh some purposive, 

process must be described. 


sa "a number o 


question. The sample must be dr 
meaningful procedure, and the’ samp 


f the infarmation received 


ay 


swith sufficient thoroughness to permit users of the measures to . 
éstimate how adequately performance on the sample tasks reflects 


performance on all possible career. skills tasks. 
r% : . 
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7 ‘EFFECTS OF VERBAL ABILITY u > -_ +. we - 
- "ON, PERFORMANCE ON MEASURES i 7 | 
‘OF CAREER SKILLS He s. o ey 
7 7 } é 


Its aégree ce validity is usually.congidered the single. most . : 
impgrtant characteristic of a test. Does the test’ measure what 4 : 
it purports to measure? Answering such a- question for measures 
(\ of.qareer skills is complicated not (only by problems of indirect - 
r _neadurénent and sampling, as discussed above, but. also by the 
fact that current: skills m@asures involve reading ability. It 
is, not pap eects for critics of various types of ‘career Gevkiopment 
; measures to ‘comment that they are in fact only measures of Yeading | ‘ 
Or. verbal ,ability. However, such a conception ° agsumes ‘that . 
reading isa well- ideveeos process invalving certain cognitive 
, activities, }he functigns of which are separate and: unique from 
Other aspect’s of cognitive activity. This. point of.view ignores 
the possibility’ that the teasoning. processes that make _ reading 
possible may be the.same reasoning processes involved in making 
3 Career decisions, and solving. other peeenee eaerieat problems 
of life. . : 


R.L. -Thorndike (1973- 1974) "renter for cons sideration. ae - 
“reading as reasoning" in an article by that.tit Thorndike 
2 developed evidence that suggested that “performance tn reading, 

‘ at least after the basic decoding skills are mastered, is ae } 

‘ primarily an indicator of the. general level of ‘the individual's / 
thinking and reasoning processes rather than a. set of distinct . 
and specialized skills." The three, lines of evidence he presented 

“were based on "(1) factorial andlyses of specific reading tasks, 

‘' (2) =the: correlations between reading tasks and both measures of: 

' general intelligence and measuring of later’ academic progress, and 
(3) the stability of difficulty in reading test items under 

- translations from. one language tm another" (p. 133). ¢ 


Neel 


He cited an article prepared a half century earlier by E.L. Thorndike 
917) in penesh the following was asserted: a 
‘Reading is a very subdue motedute, involving 
a weighing of each of many elements in a sentence, | > 
their organization in the proper relation one to ( 
another, the “selection of certain of their connotations ‘ 
’ and the rejeotion of others, and the cooperation of 
‘ Many forces:to determine final responses. Ith. Tact. «4 a 
eo thé act’of answering simple questions about a simple * 
«paragraph. ..includes all the features characteristic 
“WOE eyprcal sal (p. 323) j t 


4 


he 


-Careey skill are no. more distinct and separaqmfrom general 2 
intellectual functioning than in redding. ’ a . _ 
‘ _ —_ ’ ee -. a ; 
CAUTIONS AGAINST ua a : a oe oo 
. THE POSSIBLE’ MISUSE | — es, 3 ‘ 
OF TESTS OF CAREER SKILLS — asf! 


It seems clear that tests of ena skills" are not viewed with. 
_ sufficient ¢aution. As students wrestle with the eternal question, 
- “What shall’ I\be?", teachers and counselors. areCforever searching. 
for \the magic }tool that will help link, individuals and opportunities. 
yet fmeasuremeht for purposes of career guidance has made only modest 
progress over the past |75 years. There continues ,to be’ little 
evidence that differential aptitude tests or interest inventories : 
can provide an adequate’ basis for helping select among occupational .*" - 
alternatives or for predicting degree of’ success in an occupation. * os 


e 


e 


. 2 a 


Thus, appropriate Gaution should be observed in using tests of 
_ career skills. Users of such materials should avoid the following 
' inappropriate uses of them: mS . 


ee 
, 


(1) Never use tests of career skills alone as a basis for . 
making predictions about the future career behavior of individuals. 
Paper-and-pencjl1 tests of career-related knowledge and skills a: . 
represent at best an imprecisé sampling from an uncertaingupdverse es 
of possible knowledge and behavior. Furthermore, such tests»: 
should be regarded as status reports that help assess present 
levels of knowledge’ and.@kill. Because knowledge can be acquired 
and: skilgm"developed; wij one“knows and can do today is of " - eS, 
limited e in predicting what one will know or be able to’ do in 

.. the future. . “2. He ag ' , 


Further complicating use of sich tests in predicting future i 
behavior is the’ ‘uncertain relationship between knowing what..to \ 

do and doing it. People who behave inappropriately often do so’ 

foy reasons other than lack of knowledge., : ; 

12). Never.aasume that aj test of capeer skills te measuring 
precisely what tts title claims to be. measuring. A test is , “Ss 
considered valid only if it ifs: measuring what it purports to & 
measure. TMe constructs empl yeff in career development theory ; 
are only partially defined, and “career skills" as such probably \: 
consist of many such constructs, intertwined in.as yet unknown \ 
ways.. For example,.career- skills are probably related to general 
ability. + Yet we know Jittle jabout this relationship. 


‘ : ; : é 


Sn. 
; Bedause most tests. of: career skil ee eed ability, 
fe performance of such measures is JIkely to reflect verbal ability 
ee in large part... In fact’, wher& studies have-beén done, performance, a sf 
a -on meashres of career ayes ern elated at, moderate to high s 
,levels with performance on verbal ability meas\res? Correlation 
. studies ,/f however, simply point, ‘out ‘ degree of association;, they 


should not lead tq thg conclusion that ‘correlated skills pre, Lio: 
Hacneien. As ond colleague’ points out, shoe size and foot : . 7 
Tenge are also h ghly’ corrélated but shoes are not feet. | ‘es 


Virtually all tests are ‘coneeniiated by factors other than those. 

. they were intended, to‘meagure( Paper-and-pencil tests of career 

‘A skills require the student to read the questions, - Poor readers’ -“~ * 

eh, will perform pooriy on such tests regardless of their trué, career 

“i skills. The ,weading ability factor is probably the single’ meet 4 J” 
anponeane contaminant of career skills teste. é 


i 


ay " Be santivys tn’. the e of: norm-referenced data in.’ 


: 8 a 
as tevpreting the results of Roar er skills test adm\nistration. © §. 
, : ta abous the performance of a sfour on.a career skjlls fest ; 


reveal o ly ‘how the group performed; they tell noth¥ng about : 
how the ‘group should perform.’ Normative data help to. describe 
in cis terms; they do not evaluate. _ “ 
/ Brown (1976) poted hid connor? éssitutd to aistinguish sleaviy 
" between measyrement and eval usesons = 
4 - 
| re Measurement ariswefs. the ‘question: how much? 
: . That is, measurement provides a description of a ; 
f ' person'siperformance; it says no i bout the / rn _ 
worth or value of the pectomnancd, slg when , —_ 


we interpret a person's ha rma me ‘meee! = 4 
place some value or worth me it. A is point / 
we are) going beyond description, W tht attempting : 
to suave .the question: how goad? _/This is evalua- . 

( 


tion. p. 12) 


* .”™ 


\ | » 
Be particularly Gautious in Lneeroreting your school! Ss p rformance 
on a given test of career skills aga t any tables of sa-called 
national norms. Natignal norms, for I competency area like career 
skills are difficult - inte t because the schools selected 

for the norming sample may vary considerably in their emphasis on 
acquisition of career qpoeers . 


. « 
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iii 2, 
i ‘ 4 
dhe. bd - = a 
ey ‘ = \ 
ray ; . “4 : | | : 
* « (4) ‘Remember ‘that any career wkille test provides Only a - = i 
NO test. can,evet give a'cempletely accurate picture of a pergefi's “ 


skills*and. knowledge. People change’: New experiences and learning oy 
may result in enhanced skills. The initial sample: of questions oe 
- may have: been inappropria¥e,, and the individual might perform ~ 
quite differently on another sample of ‘questions at'a different oS oe 
De time. & : . : Ot se ae : e - : _ \ ; . a , 
° - . | - 4 og fe * . et: 
> (5) Bk afare of the riek-involved-in'making predtctione ed ~ 
based on. th& test scores of an tndivtdual. Prediction is usually | : 
based on the welationship or dggree of assoc¥ation between two 
sets of data,\e.g., scholastic aptitude test scores and freshman ,' . 
- grade-poift av ages. Many studies of this ®ort report vorrela-| 
tion coefficients in the vicinity of .45. According to Franzblay | 
(1958), "i. .cbefficientg ‘below ~40 do not. yield ‘a rediction which 
ig 25 percent better. than hace or Na yield a prediction which 
) 


‘gambling of an indtvtdual's akille at‘a particular point in times “°°. 1. 


25 percent better #han a chahce or Mandom guess, the correlat 
must be at least .66" (p. 88)%° As Barry and Wolf (1962) point 
out, "the inherent errors of the test itself, the inconclusiveness 

of the relationship between the predictor and what is being predicted 
aie it virtyally impossible for the Sounselor to predict a student's 
’ performance with, any degree of surety." (pp. 35-36) : 


: : ‘ ° i 
(6) Never use the results of performance on career skille [ 

measures to evaluate teachér or counselor effectivenessy Per for- 
mance qn educational tests:has sometfmes bee used as a factor. a ° 
in (rating the effectiveness of teachers or ,cbunselors. We believe 
that using career skills measures for such a purpose is inappro- 
priate ‘because performance,on jsuch measures is influenced by 
many factors‘pther than stat VT acetcns a Students in different 
classrooms wilt often have substantially different experiences . _, 7 
and characteristics. Parents' educational level, family income, ~— -" 
experience in part-time jobs, and the, like may influence the : A 
attainment of career skills and knowledge.g Controlling” for such . _ o- 
variables in éxperimental settings is usually guite difficult. 


ea @ 


(7) Avgtd over-interpreting the resutts Af career skills 
“teste with etaff, students, parents, and the cojnmanity in general. 
' Test scores in general have an aura of exactness about them that 
is unwarranted.’ They are simply estimates of what.a person can 
do on a limited sample of exercises or activities at a particular 
time. Their spurious appeagance of being factmal needs to be 
countered by. careful explanations of their limitations. 
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Changing ~- conceptions. ‘of cheese have. 
eee ete of new types‘of méasttres for use in 

nd. career education pr grams. , To make effecti 
assessment. matemjals, you ‘should Mave your purp 
mind and éValuate available. matergjals against t 


~ 
bart 


led toward the 
career idance 
ve ude. of such . 
oses Clearly in 
hose, purposes. 


/ Finally, remember available areer sKills’ meas es are/. 
direct measures that. amg impprfectly~from the varied universe. 
f knowledge and skill¢ that collectively can be terfi@d “eareer 


skills."| Because such papér-and-pencil. measure 


sr@quire basic, ~ . 


reading ability, erformance.on such measures may be-.di eotly.. 


‘@ influenced by verbal pbi ity. ‘Because such mea 


surge pos (sess 


+significant limitations? you priate intoge appropriate: eae 


in ee use and avoid inappropriat 


result * ' ee 
a. . Ms i a a. 
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